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Our  investigation  of  the  evolution  of  a  cell  in  this  mean  field 
network  indicates  that  many  of  the  results  on  the  existence  and 
stability  of  fixed  points  that  have  been  obtained  previously  in 
the  single  cell  theory  are  successfully  generalized  here.  In 
addition,  we  have  been  able  to  make  explicit  further  statements 
concerning  the  independent  effects  of  excitatory  and  inhibitory 
neurons  on  selectivity  and  ocular  dominance. 
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During  the  period  covered  by  this  grant,  January  1,1985  - 
December  31,  1987,  we  have  made  substantial  progress  in  our 
analysis  of  the  development  and  organization  of  neural  networks 
in  a  number  of  related  areas. 

/ 

Previously,  we  had  proposed  a  single  cell  theory  for  the 
development  of  selectivity  and  ocular  dominance  in  a  neural 
network'.1  2  During  the  contract  period,  this  theory  has  been 
generalized  to  incorporate  more  realistic  neural  networks  that 
approximate  the  actual  anatomy  of  small  regions  of  cortex.  In 
particular,  we-  have  analyzed  a  network  consisting  of  excitatory 
and  inhibitory  cells,  both  of  which  may  receive  information  from 
LGN.  These  two  cortical  cell  types  then  interact  through 
intracortical  connections  that  are  either  excitatory  or  inhibitory. 
Our  investigation  of  the  evolution  of  a  cell  in  this  mean  field 
network  indicates  that  many  of  the  results  on  the  existence  and 
stability  of  fixed  points  that  have  been  obtained  previously  in 
the  single  cell  theory  are  successfully  generalized  here.  In 
^addition,  we-have  been  able  to  make  explicit  further  statements 
concerning  the  independent  effects  of  excitatory  and  inhibitory 
neurons  on  selectivity  and  ocular  dominance.  For  example, 
shutting  off  inhibitory  cells  lessens  selectivity  and  alters  ocular 
dominance  (masked  synapses).  These  inhibitory  cells  may  be 
selective,  but  there  is  no  theoretical  necessity  that  they  be  so. 
Further,  the  intracortical  inhibitory  synapses  do  not  have  to  be 
very  responsive  to  visual  experience.  Most  of  the  learning 
process  can  occur  among  the  excitatory  LGN-cortical  synapses. 
The  theoretical  results  are  compared  with  experiment  in 
completed  and  continuing  work.  5*5-$  - 

Another  area  in  which  we  have  made  substantial  progress  is  the 
analysis  of  relaxation  mechanisms  in  neural  network.  We  have 
investigated  the  improvement  of  the  storage  efficiencey  in  the 
Hopfield  model.  The  original  model  was  capable  of  accurate 
storage  and  retrieval,  with  some  error  correction,  for  up  to  0.1N 
stored  states,  where  N  is  the  number  of  neurons.  With 
"unlearning”,  it  is  possible  for  the  system  to  recall  up  to  N  stored 
states  with  some  error  correcting  ability.7  We  have  studied  a 
variation  of  Hopfield’s  "unlearning"  which  is  able  to  achieve 
storage  and  retrieval  of  well  above  N  patterns  but  with  little 
error-correcting  capability;  this  variation  bears  a  resemblance  to 
the  Widrow-Hoff  error  correcting  procedure.  8  Our  theoretical 
analysis  of  this  scheme  showed  that  the  method  was  equivalent 


to  an  "effective  orthogonalization";  further,  the  limitations  of  this 
approach  were  analyzed  in  terms  of  an  attractor  signal  and  noise 
degradation  due  to  such  causes  as  the  interaction  of  the  system 
state  with  other  stored  states.9 

Our  analysis  of  the  Hopfield  Model  led  us  to  examine  the  general 
requirements  for  storage  systems  which  have  high  capacity  as 
well  as  good  error-correcting  capability.  This  has  led  us  to 
propose  an  alternative  relaxation  procedure  for  a  High  Density 
Storage  Model.  The  model  incorporates  an  N-dimensional 
generalized  coulombic  potential  with  essentially  inifinte  storage; 
well-defined  domains  of  attraction  can  be  specified  for  the 
stored  states,  enabling  categorization  (generalization)  of  input 
states  which  have  not  been  stored  in  the  system.1011 

Like  Neural  circuits,  the  immune  system  network  can  learn  and 
remember.  The  retrovirus  HTLVIII/LAV  which  has  been 
implicated  as  the  agent  responsible  for  the  acquired  immune 
deficiency  syndrome(AIDS)  attacks  T4 (helper)  cells  of  the 
immune  system.  We  have  shown  that  remarkable  interactions 
with  other  infections  as  well  as  strong  virus  concentration 
dependence  are  general  properties  of  ISRV.  These  theoretical 
results  have  been  compared  with  experimental  facts.12  In 
addition,  our  theoretical  model  has  been  expanded  to  include 
some  of  the  latest  clinical  results.13 


The  capacity  of  model  neural  networks  to  generalize  from  a 
partial  set  of  information  is  an  area  of  much  current  interest. 

The  ability  to  categorize  input,  to  make  generalizations  based  on 
a  limited  set  of  information,  is  one  of  the  hallmarks  of  higher 
cognitive  processes.  In  our  research  in  the  recent  past,  we  have 
undertaken  an  investigation  of  the  recall  and  generalization 
capabilities  of  a  number  of  neural  models  of  current  interest. 

In  this  context,  we  have  studied  the  Backward  Propagation  of 
Error  Model  first  proposed  by  Rummelhart  (1986).  We  have 
found  that  the  model  is  capable  of  generalizing  reasonably  well 
in  continous  spaces  where  the  decision  boundaries  are  nonlinear 
and  decision  spaces  may  not  be  simply-connected(strong 
topology).  As  a  paradigm,  we  have  considered  the  problem  of 
learning  to  distinguish  the  two  regions  enclosed  by  two 
concentric  circles:  the  outer  annulus,  with  target  ouput  of  1,  and 
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the  inner  disk,  with  target  output  of  0.  In  simulations  of  this 
problem  using  the  Backward  Propagation  Network,  we  have 
found  that  after  training  with  randomly  selected  patterns  in  the 
two  regions,  the  generalization  curve  of  output  versus  radius 
approaches  that  of  the  ideal  step-function,  provided  a  sufficient 
number  of  patterns  is  used  and  provided  that  we  do  not  saturate 
the  network  with  too  many  training  patterns. 

In  contrast,  for  a  difficult  problem  on  a  lattice  such  as  the  parity 
problem  with  highly  discontinous  decision  boundaries,  patterns 
which  are  not  used  for  training  will  generally  not  be  correctly 
identified.  In  fact,  the  network  will  identify  them  as  the  state  of 
opposite  parity.  Our  analysis  of  continous  space  problems  as 
described  above,  suggests  in  this  instance  that  the  system  may 
be  trying  to  generalize  based  on  the  nearest  available 
information  in  the  continous  space  sense,  which  in  the  case  of 
the  parity  problem,  where  nearest  neighbor  states  have  opposite 
parity,  leads  to  the  misidentification. 

Currently,  we  are  exploring  ways  to  modify  the  original 
Backward  Propagation  alogrithm  so  that  it  will  be  useful  for 
generalization  problems  like  that  of  parity,  where  generalization 
in  the  continuous-space  sense,  that  is,  assigning  an  output  that 
mimics  that  of  the  nearest  data  point,  is  not  an  adequate 
solution.  A  number  of  possibilities  are  being  investigated. 
Additional  symmetry-sorting  terms  added  to  the  Backward 
Propagation  energy  functional  may  prove  useful  in  developing 
networks  which  can  handle  generalization  tasks  like  parity. 
Another  possibility  is  to  present  the  data  to  the  network  in  such 
a  way  that  it  is  forced  to  satisfy  the  constraints  of  more  than  one 
pattern  at  the  same  time;  the  original  Backward  Propagation 
algorithm  examines  only  one  pattern  at  a  time.  In  this  context, 
additional  symmetry-sorting  energy  terms  may  be  also  quite 
useful.14 


1  Theory  for  the  Development  of  Neuron  Selectivity:  Orientation 
Specificity  and  Binocular  Interaction  in  Visual  Cortex,  Bienenstock,  E.  L., 
Cooper,  L.  N,  and  Munro,  P.,  Journal  of  Neuroscience  2.  32-48  (1982). 

distributed  Memory,  Cooper,  L.  N.,  in  Encyclopedia  of  Neuroscienccs,  Vol. 
II,  ed.  G.  Adelman  pp.  633-634,  Birkhauser,  Boston,  1987;  also  ARO 
Technical  Report,  March  13,  1985. 


^Cortical  Plasticity:  Theoretical  Analysis,  Experimental  Results,  in 
Imprinting  and  Cortical  Plasticity,  ed.  J.  P.  Rauschecker  and  P.  Marler, 
John  Wiley  &  Sons,  N.  Y..  1987,  also  ARO  Technical  Report,  July  10,  1985. 
4Local  and  Global  Factors  in  Learning,  to  be  published  in  Brain  Structure, 
Learning  and  Memory,  ed.  J.  Davis,  R.W.  Newburgh,  and  E.  Wegmen,  by 
AAAS,  also  ARO  Technical  Report,  November  6,  1985. 

5  Mean  Field  Theory  of  a  Neural  Network,  Cooper,  L.  N,  Scofield,  C.  L.,  to  be 
published  in  Proc.  Natl.  Acad.  Sci.,  also  ARO  Technical  Report,  January, 
1988. 

^Connectivity  in  Neural  Networks,  Cooper,  L.  N,  to  be  published  in  the 
Proceedings  of  the  Copenhagen  Conference  on  Computer  Simulation  in 
Brain  Research. 

7Storing  and  Retrieving  Data  in  a  Parallel  Distributed  Memory  System, 
Potter,  T.  W.,  Dissertation  and  ARO  Technical  Report,  June  9,  1987. 

8op.  cit. 

9The  Hopfield  Model  and  Beyond,  Bachmann,  C.  M.,  ARO  Technical  Report, 
December  15,  1986. 

10A  Relaxation  Model  for  Memory  with  High  Storage  Density,  Bachmann, 
C.  M.,  Cooper,  L.  N,  Dembo,  A.,  and  Zeitouni,  O.,  Proc.  Natl.  Acad.  Sci.,  Vol.  84, 
pp.  7529-7531,  November,  1987,  also  ARO  Technical  Report,  June  22,  1987. 
^General  Potential  Surfaces  and  Neural  Networks,  Dembo,  A,  Zeitouni,  O., 
to  be  published  in  Phys.  Rev.  A,  also  ARO  Technical  Report,  June  24,  1987. 
12Theory  of  an  immune  system  retrovirus.  Cooper,  L.  N,  Proc.  Natl.  Acad. 
Sci.,  Vol.  83,  pp.9159-9163,  December  1986. 

13 Analysis  of  Immune  System  Retrovirus  Equations,  Intrator,  N., 
Deocampo,  G.  P..  Cooper,  L.  N.,  (to  be  published  in  the  Conference  Record 
of  The  Santa  Fe  Institute,  July  1987). 

^Generalization  and  the  Backward  Propagation  Neural  Network, 
Bachmann,  C.  M.,  ARO  Technical  Report,  January,  1988. 


An  account  is  given  of  the  biological  basis  for  and  the  elementary 
properties  of  distributed  memory.  It  is  shown  how  a  neural  network 
can  be  constructed  to  contain  such  a  memory.  Also  discussed  are 
problems  of  address  and  recall  of  stored  items,  as  well  as  the 
possible  sites  of  storage  of  short  and  long-term  memory. 


An  account  is  given  of  a  theory  of  and  experimental  results  on  develop¬ 
ment  and  modification  of  selectivity  and  ocular  dominance  in  visual 
cortex.  The  single  cell  theory  is  generalized  to  be  applicable  to  a 
neural  network.  Also  discussed,  in  the  context  of  the  theoretical 
ideas,  are  experiments  on  the  modifiability  of  inhibitory  cells  and  on 
possible  candidates  for  global  controllers  of  learning. 


Like  neural  circuits,  the  immune  system  network  can  learn  and  re' 
member.  The  retrovirus  HTLVIII/LAV  which  has  been  implicated  as 
the  agent  responsible  for  the  acquired  immune  deficiency  syn¬ 
drome  (AIDS)  attacks  T^(helper)  cells  of  the  immune  system.  In 
this  paper,  we  contrast  the  growth  of  a  'normal'  virus  with  what 
we  call  an  Immune  System  Retrovirus  (ISRV):  a  retrovirus  that 
attacks  ^(helper)  cells  of  the  immune  system.  We  show  that 
remarkable  interactions  with  other  infections  as  well  as  strong 
virus  concentration  dependence  are  general  properties  of  ISRV. 
These  ideas  are  compared  with  some  known  experimental  facts. 


Connectivity  in  Neural  Networks 
ABSTRACT 

A  single  cell  theory  for  the  development  of  selectivity  and  ocular 
dominance  in  visual  cortex  has  been  generalized  to  incorporate  more 
realistic  neural  networks  that  approximate  the  actual  anatomy  of  small 
regions  of  cortex.  In  particular  we  have  analyzed  a  network  consisting 
of  excitatory  and  inhibitory  cells,  both  of  which  may  receive  informa¬ 
tion  from  LGN.  These. two  cortical  cell  types  then  interact  through 
intracortical  connections  that  are  either  excitatory  or  inhibitory. 

Our  investigation  of  the  evolution  of  a  cell  in  this  mean  field  network 
indicates  that  many  of  the  results  on  existence  and  stability  of  fixed 
points  that  have  been  obtained  previously  in  the  single  cell  theory  can 
be  successfully  generalized  here.  We  can,  in  addition,  make  explicit 
further  statements  concerning  the  independent  effects  of  excitatory  and 
inhibitory  neurons  on  selectivity  and  ocular  dominance.  For  example, 
shutting  off  inhibitory  cells  lessens  selectivity  and  alters  ocular 
dominance  (masked  synapses).  These  inhibitory  cells  may  be  selective 
but  tnere  is  no  theoretical  necessity  that  they  be  so.  Further  the 
intracortical  inhibitory  synapses  do  not  have  to  be  very  responsive  to 
visual  experience.  Most  of  the  learning  process  can  occur  among  the 
excitatory  LGN-cortical  synapses.  Some  of  these  theoretical  ideas  are 
compared  with  experimental  results. 


Mean  Field  Theory  of  a  Neural  Network 

(Visual  Cortex/Synaptic  Modification) 

Leon  N  Cooper  and  Christopher  L.  Scofield* 

Center  for  Neural  Science 
and  Physics  Department 
Brown  University 
Providence,  Rhode  Island  02912 

ABSTRACT 

A  single  cell  theory  for  the  development  of  selectivity  and  ocular  dominance  in  visual  cortex 
has  been  generalized  to  incorporate  more  realistic  neural  networks  that  approximate  the 
actual  anatomy  of  small  regions  of  cortex.  In  particular  we  have  analyzed  a  network 
consisting  of  excitatory  and  inhibitory  cells,  both  of  which  may  receive  information  from 
LGN  and  then  interact  through  cortico-cortical  synapses  in  a  mean  field  approximation.  Our 
investigation  of  the  evolution  of  a  cell  in  this  mean  field  network  indicates  that  many  of  the 
results  on  existence  and  stability  of  fixed  points  that  have  been  obtained  previously  in  the 
single  cell  theory  can  be  successfully  generalized  here.  We  can,  in  addition,  make  explicit 
further  statements  concerning  the  independent  effects  of  excitatory  and  inhibitory  neurons 
on  selectivity  and  ocular  dominance.  For  example,  shutting  off  inhibitory  cells  lessens 
selectivity  and  alters  ocular  dominance,  (masked  synapses).  These  inhibitory  cells  may  be 
selective  but  there  is  no  theoretical  necessity  that  they  be  so.  Further  the  intracortical 
inhibitory  synapses  do  not  have  to  be  very  responsive  to  visual  experience.  Most  of  the 
learning  process  can  occur  among  the  excitatory  LGN-cortical  synapses.  Some  of  these 
ideas  are  compared  with  experimental  results. 

♦Present  address:  Nestor,  Inc.,  1  Richmond  Square,  Providence,  Rhode  Island,  02906. 


STORING  &  RETRIEVING  DATA  IN  A  PARALLEL  DISTRIBUTED 

MEMORY  SYSTEM 

ABSTRACT 

TERRY  WAYNE  POTTER 

B.S.,  Math,  Kent  State  University,  1970 
M.S.,  Computer  Science,  Rutgers  University,  1972 

The  storage  and  retrieval  of  patterns  in  a  Hopfield-like  Parallel  Distributed  Mem- 
ory  is  investigated  experimentally  with  a  view  toward  increasing  its  storage  capacity. 

The  first  two  Chapters  give  an  overview  of  distributed  memories  and  in  particular 
the  Hopfield  distributed  memory.  This  is  followed  by  a  Chapter  which  experimen¬ 
tally  identifies  the  basic  storage  capacity  of  the  original  Hopfield  memory  when 
using  text  patterns. 

.This  dissertation  then  experimentally  investigates  new  and  untested  methods  to 
increase  the  storage  capabilities  of  a  Hopfield-like  neural  net.  Increasing  the  storage 
capacity  by  using  the  continuous-valued  Hopfield  memory  is  explored  in  Chapter  3 
and  the  impact  on  capacity  of  data  representation  is  experimentally  investigated  in 
Chapter  4.  We  then  focus  on  new  ways  of  storing  data  (changing  the  interconnect 
strengths)  including  in  Chapter  7  developing  a  new  method  called  Modifying  the 
Energy  Contour  or  MEC.  In  addition,  this  Chapter  also  outlines  how  to  increase 
error-tolerance  through  the  use  of  noisy  patterns. 

The  Hopfield  distributed  memory  is  then  contrasted  to  another  intelligent  mem¬ 
ory  subsystem  based  on  more  of  a  traditional  computer  technology.  In  Chapter  8 
we  see  that  traditional  computer  technology  using  data-parallel  techniques  has  a 
greater  storage  efficiency  than  possible  with  current  Hopfield-like  distributed  mem¬ 
ories.  The  design  of  this  data-parallel  memory  is  based  in  part  on  what  is  learned 
experimentally  from  the  preceding  Chapters  on  the  Hopfield-like  distributed  mem- 


ory.  This  fast  data-parallei  approach  also  supports  retrieval  of  data  patterns  with 
noisy  inputs  although  it  does  not  have  all  the  functionality  of  the  Hopficld-like 
distributed  memory. 


Abstract 


The  standard  Hopfield  model  (both  digital  and  analog) 
and  algorithms  to  improve  its  performance  are  reviewed. 
An  analysis  of  the  model  and  the  modification  algorithms 
is  given.  Future  directions  for  continuous  models  which 
have  both  large  capacity  and  good  error-correcting  capa¬ 
bilities  are  examined. 
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Abstract 


We  present  a  relaxation  model  based  on  an  N-dimensional  Coulomb  poten¬ 
tial.  The  model  has  essentially  infinite  storage  capacity  and,  in  addition, 
well-defined  basins  of  attraction  about  stored  memory  states.  The  model 
is  compared  with  the  Hopfield  relaxation  model. 
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ABSTRACT 

Investigating  Hopfield's  model  of  associative  memory  implementation  by  a 
neural  network,  led  to  a  generalized  potential  system  with  a  much  superior  per¬ 
formance  as  an  associative  memory.  In  particular,  there  are  no  spourious  memories, 
and  any  set  of  desired  points  can  be  stored,  with  unlimited  capacity  (in  the  con¬ 
tinuous  time  and  real  space  version  of  the  model) .  There  are  no  limit  cycles 
in  this  system,  and  the  size  of  all  basins  of  attraction  can  reach  up  to  half 
the  distance  between  the  stored  points,  by  proper  choice  of  the  design  parameters. 

f 

A  discrete  time  version  with  its  state  space  being  the  unit  hypercube 
is  also  derived,  and  admits  superior  properties  compared  to  the  corresponding 
Hopfield  network.  In  particular  the  capacity  of  any  system  of  N  neurons,  with  a 
fixed  desired  size  of  basins  of  attractions,  is  exponentially  growing  with  N  and 
is  asymptotically  optimal  in  the  information  theory  sense.  The  computational 
complexity  of  this  model  is  slightly  larger  than  that  of  the  Hopfield  memory, 
but  of  the  same  order. 

The  results  are  derived  under  an  axiomatic  approach  which  determines  the 
desired  properties  and  shows  that  the  above  mentioned  model  is  the  only  one 


to  achieve  them. 
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Abstract 

An  attempt  has  been  made  to  mathematically  model  the  patterns  of  viral  growth 
and  the  associated  immune  system  response  characteristically  associated  with 
infection  by  the  human  immunodeficiency  virus  (HIV).  By  employing  a  very 
simple  model  of  viral  growth  and  the  humoral  immune  system  response,  the 
interaction  between  a  'normal'  virus  and  the  immune  system  response  is  compared 
with  that  of  a  viral  entity  called  an  immune  system  retrovirus  (ISRV)  (Cooper, 
1986).  Some  of  the  consequences  of  the  model  are  presented  in  numerical 
simulations  and  compared  to  other  models  treating  the  cellular  response  of  the 
immune  system . 
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ABSTRACT 


Results  of  simulations  in  discrete  and  continuous  input 
simulations  are  discussed  for  the  Rumelhart's  Backward 
Propagation  of  Error  Neural  Network.  Comparison  of.  the 
results  offers  a  way  to  understand  the  problem  of  generalization 
from  a  partial  set  of  data  in  the  Backward  Propagation  Network 


